Classification algorithms based on on-line learning, such as artificial neural networks, have the ability to abstract relationships between inputs and their corresponding classification labels in an incremental fashion. As more points of the input/output mapping are sampled, the algorithm is capable of creating decision boundaries that separate the various classes in a non-linear fashion. An important class of on-line learning algorithms is based on adaptive resonance theory (ART). ART-based learning systems are capable of on-line learning and classification of both binary and analog inputs. Several variants of ART systems can be found in the literature such as Fuzzy-ARTMAP, PROBART, and Gaussian ARTMAP. These systems are capable of on-line supervised learning. The main drawback with these algorithms is their sensitivity to noise in the training data that leads to the creation of a prohibitively large number of rules for classification. Efforts have been undertaken to develop techniques to minimize sensitivity to noise as well as to improve the ratio between accuracy of prediction to the number of rules required for classification. In addition to their noise sensitivity, ART-based learning systems are not amenable to implementation in hardware, as several additional steps must be taken in order to extract rules that can be put into a fuzzy inference form. Because of this, ART-based learning systems are generally implemented as software, which is substantially slower than an analogous hardware system would be. This is also compounded by the fact that these algorithms tend to generate a large number of classification rules.
Another class of on-line learning algorithms is based on the extraction of knowledge in the form of fuzzy rules by tuning the parameters of a fuzzy logic inference network. Traditionally, fuzzy systems consist of a set of fuzzy IF-THEN rules that are derived based on knowledge of a domain expert. Inferences on the output for a new input are then made based on these rules using the fuzzy inference network. This approach is, however, based on heuristic observation of the system by the expert to extract the appropriate rules. To overcome this deficiency, there are several approaches aimed at deriving fuzzy IF-THEN rules directly from numerical data observed from the system. Predominantly, these approaches depend on optimization of fuzzy system parameters in an off-line fashion from numerical data to obtain the fuzzy rules. Thus, these systems are incapable of incremental learning.
Another class of learning systems is the fuzzy inference network, an example of which is the self-constructing fuzzy inference network (SONFIN). The SONFIN is capable of deriving a fuzzy rule base in a self-organized and on-line fashion from numerical data. Since the SONFIN architecture is designed based on fuzzy inference systems, the network can make inferences on any given input data based on its rule base at any given time. This makes the SONFIN an attractive network for many real-time applications where the environment is changing dynamically and yet there is a need to abstract knowledge from the system in the form of fuzzy rules. The SONFIN performs very well for classification/functional mapping of low-dimensional input data. However, when the dimensions of the input space increases (such as where the number of input features exceeds 10), the algorithm is ineffective for learning because of a problem associated with the learning rule. The inability to perform adequate learning essentially converts the SONFIN into a poor clustering algorithm, thus leading to poor learning and prediction capabilities as well as a larger number of fuzzy rules.
A flow diagram depicting the operation of the SONFIN is provided in FIG. 1. A portion of the flow diagram represents a generic fuzzy inference network 100. The SONFIN provides the adaptations necessary to provide on-line learning. In an inputting step 102, an N-dimensional input pattern is provided to the fuzzy inference network 100. The fuzzy inference network 100 then computes the membership values for each input dimension for each rule in a membership value computation step 104. The firing strength of the rules is determined based on the input and is checked to determine whether it exceeds a predetermined threshold in a firing strength checking step 106. If the firing strength exceeds the threshold, then the fuzzy inference network 100 computes normalized rule strengths for the rule in a normalizing step 108. The fuzzy inference network 100 then computes an output using centroid defuzzification in a defuzzifying step 110. In the steady-state operation of a trained network, the result of the defuzzifying step 110 is the output of the fuzzy inference network 100. In the SONFIN, the output of the fuzzy inference network 100 is provided to a back-propagation algorithm where the rule parameters are updated in a back-propagating step 112. A check is made to determine whether there are more inputs, and the cycle begins again. If the firing strength of the rules was less than the threshold, and if the rule does not satisfy a fuzzy similarity measure, then a new rule is created with new membership functions along each input dimension in a rule-creating step 114.
Although SONFIN provides a self-organized and on-line learning system, it suffers from a major drawback because its performance is dependent on the number of input dimensions. Thus, SONFIN is effectively useless for on-line classification of high-dimensional data such as that occurring in applications such as vehicle occupant sensing, weather forecasting, and stock market analysis/economic forecasting.
It is therefore desirable to provide a self-organized, on-line learning system, the performance of which is independent of the number of input dimensions. Because such a system would be capable of elucidating its learned knowledge in the form of fuzzy rules, the system can be evaluated with new data using those rules without any delays, thus saving considerable time and data collection effort while developing a learning system. Another advantage is that if increasing the number of input dimensions increased the number of features, there would be no need for crafting the parameters of the network and the system would not suffer in its learning ability. Thus, the system would be more robust and flexible for evaluating different classification strategies.
References of interest relative to neural networks and their use in classification involving high-dimensional problems include the following:
    1. Juang, C. F., and Lin, C. T., “Δn On-Line Self-Constructing Neural Fuzzy Inference Network and Its Application,” IEEE Transactions on Fuzzy Systems, vol. 6, no. 1, pp. 12-32, February 1998.    2. Carpenter, G. A., and Grossberg, S., “The ART of adaptive pattern recognition by a self-organizing neural network,” Computer, vol. 21, pp. 77-88, 1988.    3. Carpenter, G. A., Grossberg, S., Markuzon, N., Reynolds, J. H., Rosen, D. B., “Fuzzy ARTMAP: A Neural Network Architecture for Incremental Supervised Learning of Analog Multidimensional Maps,” IEEE Transactions on Neural Networks, vol. 3, no. 5, pp. 698-712, September 1992.    4. Marriott S., and Harrison, R. F., “A modified fuzzy ARTMAP architecture for approximation of noisy mappings,” Neural Networks, vol. 2, pp. 359-366, 1995.    5. Williamson, J. R., “Gaussian ARTMAP: A neural network for fast incremental learning of noisy multidimensional maps,” Neural Networks, vol. 9, pp. 881-997, 1996.    6. Srinivasa, N., “Learning and generalization of Noisy Mappings Using a Modified PROBART Neural Network,” IEEE Transactions on Signal Processing, vol. 45, no. 10, pp. 2533-2550, October 1997.    7. Lee, C. C., “Fuzzy Logic in control systems: Fuzzy logic controller—Part II,” IEEE Transactions on Systems, Man and Cybernetics, vol. 15, pp. 419-435, March/April 1990.    8. Jang, J. S., “ANFIS: Adaptive-network-based fuzzy inference system,” IEEE Transactions on Systems, Man and Cybernetics, vol. 23, pp. 665-685, May 1993.    9. Takagi, T. and Seguno, M., “Fuzzy identification of systems and its applications to modeling and control,” IEEE Transactions on Systems, Man and Cybernetics, vol. 15, pp. 116-132, January 1985.    10. Wang, L. X., and Mendel, J. M., “Generating fuzzy rules by learning from examples,” IEEE Transactions on systems, Man and Cybernetics, vol. 22, no. 6, pp. 1414-1427, Nov./Dec. 1992.    11. Beyer, K., Goldstein, J., Ramakrishnan, R., and Shaft, U., “When is Nearest Neighbor Meaningful,” Proc. Of Seventh International Conference on Database Theory, Jerusalem, Israel, 1999.